aft  HIE  COB  AD-A178  521 


Navy  Personnel  Research  and  Development  Center 

Ian  Pt—o,CA  Will  WOO  NPR3CTR  87-19  March  1987 


\ 


The  Course  Evaluation  System 


Approwd  for  public  wtim;  distribution  is  unlimttad. 


SL 

E 

W 


DTIC 

ELECTE 
MAR  3  01987 


»  I 


87  8  80  04 


NPRDC  TR  87-19 


March  1987 


The  Course  Evaluation  System 


John  A.  Ellis,  Ph.  D. 
Frederick  G.  Knirk,  Ph.  D. 
Barbara  E.  Taylor 
Barbara  A.  McDonald,  Ph.  D. 


Reviewed  by 

Nick  H.  YanMatre,  Ph.  D. 


Approved  by 
Edwin  G.  Aiken,  Ph.  D. 


Released  by 
B.  E.  Bacon 
Captain,  U.  S.  Navy 
Commanding  Officer 


and 


J.  S.  McMichael,  Ph.  D. 
Technical  Director 


Approved  for  public  release; 
distribution  is  unlimited. 


Navy  Personnel  Research  and  Development  Center 
San  Diego,  California  92152-6800 


UNCLASSIFIED 


/)/)'///_,■  t-6 ->  / 


REPORT  DOCUMENTATION  PAGE 

1*  REPORT  SECURITY  CLASSIFICATION 

UNCLASSIFIED 

lb  RESTRICTIVE  MARKINGS 

2a  SECURITY  CLASSIFICATION  AUTHORITY 

3  DISTRIBUTION  /AVAILABILITY  OF  REPORT 

Approved  for  public  release;  distribution  is 

2d  DECLASSIFICATION  /DOWNGRADING  SCHEDULE 

unlimited. 

4  PERFORMING  ORGANIZATION  REPORT  NUMBER(S) 

NPRDC  TR  87-19 

5  MONITORING  ORGANIZATION  REPORT  NUMBER(S) 

6a  NAME  OF  PERFORMING  ORGANIZATION 

Navy  Personnel  Research  and 
Development  Center 

6b  OFFICE  SYMBOL 
(If  applicable) 

Code  51 

7a.  NAME  0*  MONITORING  ORGANIZATION 

i 

6c  ADDRESS  (Oty.  State,  and  ZIP  Code) 

San  Diego,  CA  92152-6800 

7b  ADDRESS  (C/ty,  State,  and  HP  Code) 

8a  NAME  OF  FUNDING  / SPONSORING 
ORGANIZATION 

8D  OFFICE  SYMBOL 
(if  applicable) 

9  PROCUREMENT  INSTRUMENT  IDENTIFICATION  NUMBER 

PROGRAM 


PROJECT 


ELEMENT  NO  I  NO 


62233 


NP2A 


WORK  UNIT 
ACCESSION  NO 


M  Ti T i. E  ( include  Security  Classification) 

The  Course  Evaluation  System 


12  PERSONA.  AUThOR(S) 

Ellis,  John;  Knirk,  Frederick;  Taylor,  Barbara;  and  McDonald,  Barbara 


’Mtffi£fi‘,EPORT 


UbJE^p^SeQct  |15  SUP6  COUNT 


16  SUPPLEMENTARY  NOTATION 


COSATI  COOES 


GROUP  SUB-GROUP 


18  SUBJECT  TERMS  (Continue  on  reverse  if  necessary  and  identify  by  block  number) 

Instructional  system  development, Equality  control,  quality 
assurance ^ _ - 


19  ABSTRACT  (Continue  on  reverse  if  necessary  and  identify  by  block  number) 

'  >In  the  face  of  increasing  pressures  on  the  training  system  to  increase  its  efficiency  and 
effectiveness,  the  improvement  of  Navy  classroom  training  is  an  important  issue.  This  report  describes 
the  Course  Evaluation  System  (CES)  which  is  a  prescriptive  methodology  for  evaluating  the  quality  of 
classroom  training.  It  is  designed  to  give  course  managers  and  instructors  the  capability  to  pinpoint 
problems  in  ineffective  courses  and  make  effective  revisions.  This  will  result  in  more  efficient  and 
effective  training.  It  is  based  on  the  principles  of  systematic  instructional  development  and  on 
empirical  work  demonstrating  effective  instructional  strategies  and  environments.  The  CES 
methodology  has  been  validated  in  a  number  of  applications.  It  consists  of  two  major  sections,  the 
objective-test  item  evaluation  and  the  instructional  presentation  evaluation.  Each  section  contains 
check  lists  and  detailed  instructions  for  how  to  perform  the  evaluations.  c,  '  c  i  »  V  v.  <. 


rM- 


4  it*.  I.  .  .  . 


t/.'  i 


20  Distribution  i  AVAILABILITY  Of  ABSTRACT 
OuNCLASSlflEO/UNLlMITID  □  SAME  AS  RPT 


22*  NAME  Of  RESPONSIBLE  INDIVIDUAL 

John  A.  Ellis 


DO  FORM  1473, 84  mar 


act  21.  ABSTRACT  SECURITY  CLASSIFICATION 

as  rpt  Done  USERS  UNCLASSIFIED _ _ 


22b  telephone  (include  Area  Code)  22c.  OFFICE  SYMBOL 
(619)  225-6434  Code  51 


83  APR  edition  m»y  bt  v»ed  until  exhausted  SECURITY  CLASSIFICATION  Of  THIS 

All  other  editions  *re  obsolete  "  -  ----  -  ~ 


UNCLASSIFIED 


FOREWORD 


The  Course  Evaluation  System  (CES)  was  developed  as  part  of  the 
Classroom  Instructional  Technologies  task  under  the  Personnel,  Training,  and 
Human  Factors  Technology  (NP2A)  Block  of  the  6.2  Mission  Support  Technology 
Program  Element  62233.  The  ultimate  goal  of  the  task  is  to  provide  guidelines 
for  the  classroom  of  the  1990s.  The  CES  was  the  result  of  a  data  collection  effort 
to  determine  the  current  status  of  Navy  classroom  training.  It  is  intended  for  use 
by  Navy  Curriculum  and  Instructional  Standards  Officers,  course  managers, 
senior  instructors,  educational  specialists,  and  any  other  personnel  involved  in 
training. 


B.  E.  BACON 
Commanding  Officer 


J.  S.  McMICHAEL 
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Summary 


Problem/Background 

The  Navy  currently  teaches  over  7000  courses.  The  great  majority  of  these 
are  presented  in  traditional  classrooms  with  a  group  of  students  being  taught  by 
a  single  instructor.  Recent  directives  by  the  Chief  of  Naval  Education  and 
Training  (CNET)  and  the  Chief  of  Naval  Operations  (CNO)  indicate  that  this 
training  format  will  continue  to  play  a  predominant  role  in  Navy  education  and 
training.  Given  this  situation,  the  quality  and  effectiveness  of  Navy  classroom 
instruction  is  an  important  concern.  Recent  evaluations  of  lecture-type 
instruction  in  civilian  schools  have  shown  that  instructional  quality  is  highly 
variable,  non-standard  and  often  poor.  Similar  problems  have  been  observed  in 
Navy  schools.  In  addition,  recent  research  on  learning  in  civilian  schools  has 
shown  that  in  traditional  classroom  settings,  variables  affecting  student 
achievement  can  be  effectively  controlled.  Yet,  there  have  been  no  systematic 
attempts  to  explore  the  applicability  of  controlling  these  or  other  potentially 
useful  variables  in  Navy  classroom  training. 

Overview 

In  the  face  of  increasing  pressures  on  the  training  system  to  increase  its 
efficiency  and  effectiveness,  the  improvement  of  Navy  classroom  training  is  an 
important  issue.  This  report  presents  the  Course  Evaluation  System  (CES), 
which  is  a  prescriptive  methodology  for  evaluating  the  quality  of  classroom 
training. 

Purpose 

The  (CES)  gives  course  managers  and  instructors  the  capability  to  pinpoint 
problems  in  ineffective  courses  and  make  effective  revisions.  This  will  result  in 
more  efficient  and  effective  training. 

The  CES  is  based  on  the  principles  of  systematic  instructional  development 
and  on  empirical  work  demonstrating  effective  instructional  strategies  and 
environments.  The  CES  methodology  has  been  validated  in  a  number  (see 
references)  of  applications.  It  consists  of  two  major  sections,  the  objective  -  test 
item  evaluation  and  the  instructional  presentation  evaluation.  Each  section 
contains  check  lists  and  detailed  instructions  for  how  to  perform  the  evaluations. 

Recommendations 

CNET  is  currently  developing  a  Navy-wide  policy  for  overall  quality  control 
of  training  and  training  management.  It  is  recommended  that  the  Course 
Evaluation  System  be  incorporated  in  this  policy. 
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The  Course  Evaluation  System 


Introduction 

The  Course  Evaluation  System  (CES)  is  a  tool  for  evaluating  the 
effectiveness  and  appropriateness  of  a  course  or  a  segment  of  instruction  (for 
example,  a  module  or  lesson)  within  a  course.  The  procedures  were  designed  to  be 
used  by  Navy  Curriculum  and  Instructional  Standards  Officers  (CISOs),  course 
managers,  senior  instructors,  educational  specialists  and  any  other  personnel 
involved  in  training.  The  system  focuses  on  the  processes  of  instruction  in  the 
traditional  Navy  classroom.  The  Navy  currently  teaches  over  7000  courses.  The 
great  majority  of  these  are  presented  in  traditional  classrooms  and  laboratories 
with  a  group  of  students  being  taught  by  a  single  instructor.  Recent  directives 
by  the  Chief  of  Naval  Education  and  Training  (CNET)  and  the  Chief  of  Naval 
Operations  (CNO)  indicate  that  this  training  format  will  continue  to  play  a 
predominant  role  in  Navy  education  and  training.  Given  this  situation,  the 
quality  and  effectiveness  of  Navy  classroom  instruction  is  an  important  concern. 
Recent  evaluations  of  lecture-type  instruction  in  civilian  schools  have  shown  that 
instructional  quality  is  highly  variable,  non-standard  and  often  poor.  Similar 
problems  have  been  observed  in  Navy  schools  (Ellis,  1985;  VanKekerix,  Wulfeck, 
&  Montague,  1982).  In  addition,  research  on  learning  has  shown  that  variables 
affecting  student  achievement  can  be  effectively  controlled.  Yet,  there  have  been 
no  systematic  attempts  to  explore  the  applicability  of  controlling  these  or  other 
potentially  useful  variables  in  Navy  classroom  training. 

To  develop  the  CES,  military  and  civilian  training-related  research  studies 
(e.g.,  Bloom,  1984)  were  reviewed  for  those  variables  which  most  significantly 
affect  learning.  Current  Navy  classroom  evaluation  procedures  were  also 
reviewed.  The  CES  represents  a  synthesis  of  the  above  information  and  is  based 
on  variables  which  most  affect  the  quality  of  Navy  training.  The  basic  CES 
methodology  has  been  empirically  validated  in  a  number  of  research  and  practical 
applications  (e.g.,  Ellis,  1985;  Montague,  Ellis,  &  Wulfeck,  1983;  Stern  & 
Fredericks,  1982).  A  future  report  will  describe  an  extensive  application  of  the 
CES  to  current  Navy  classroom  training. 

The  CES  gives  course  managers  and  instructors  the  capability  to  pinpoint 
problems  in  ineffective  courses  and  make  revisions.  It  evaluates  course  quality  by 
assessing  the  adequacy  and  consistency  of  the  three  primary  components  of 
instruction;  objectives,  test  items,  and  the  instructional  presentation.  There  are 
two  evaluation  components  of  the  CES:  Objective  and  Test  Item  Adequacy  and 
Consistency  Evaluation  and  Presentation  Evaluation.  There  are  also  preliminary 
steps  involving  classification  of  objectives  to  be  evaluated  and  separating  broad 
objectives  into  smaller  units.  The  CES  user  may  choose  to  apply  either  one  or 
both  CES  components.  An  evaluation  may  involve  an  entire  course  or  be  limited 
to  specific  lesson(s)  or  module(s)  that  have  problems. 


For  objective  and  test  evaluation,  course/lesson/module  objectives  and  test 
iiems  are  examined  to  determine  their  adequacy  and  how  well  they  relate  to  the 
tasks  the  student  must  perform  on  the  job.  The  objectives  are  then  matched  to 
related  test  items  to  check  for  consistency  by  determining  if  the  conditions, 
action(s),  and  standard(s)  in  the  objectives  match  those  in  the  test  items. 

The  presentation  evaluation  involves  reviewing  the  instructor  and  student 
guides  and  observing  the  instructor’s  presentation  in  the  classroom.  Both  the 
consistency  of  the  presentation  with  the  objectives  and  the  adequacy  of  the 
presentation  are  evaluated. 

Before  applying  the  Course  Evaluation  System,  you  must  obtain  the 
necessary  resource  documents.  These  include  the  course  objectives,  a  copy  of  the 
course  tests,  instructor  guide(s)  and  student  guide(s).  Also,  you  must  determine  if 
the  entire  course  is  to  be  evaluated  or  just  specific  course  segments. 

The  next  two  sections  discuss  who  should  use  the  CES  and  when  it  should 
be  applied.  The  rest  of  this  document  describes  in  detail  how  to  use  the  CES. 
Appendix  A  contains  a  complete  CES  example  so  that  the  reader  can  see  what 
the  completed  evaluation  forms  look  like  and  how  they  can  be  used. 


Who  Should  Use  the  Course  Evaluation  System 

The  Course  Evaluation  System  was  designed  to  be  used  by  personnel 
responsible  for  course  management,  course  revision,  course  evaluation,  instructor 
evaluation,  and  maintenance  of  instructional  quality.  In  the  Navy  these  people 
include  senior  enlisted  personnel  who  are  course  instructors  and  managers,  CISOs, 
civilian  educational  specialists  and  training  specialists,  and  contractor  personnel. 
This  list  includes  people  who  are  subject  matter  experts  (SMEs)  (e.g., instructors) 
and  people  who  are  not  SMEs  (e.g. educational  specialists).  Because  of  this,  it  is 
important  to  point  out  that  the  CES  can  only  be  used  if  you  are  an  SME  or  if 
you  have  at  least  one  (preferably  two  or  three)  SMEs  available  to  assist  you  in 
your  evaluation.  In  addition,  if  you  are  an  SME  you  should  know  the  basic 
principles  of  instruction;  that  is,  you  should  have  at  least  completed  basic 
instructor  training. 


When  Should  the  Course  Evaluation  System  Be  Applied 

As  you  will  see  by  reading  the  rest  of  this  document,  using  the  CES  to 
evaluate  instruction  is  a  labor  intensive  process.  Applying  the  CES  to  an  entire  6 
week  course  would  be  a  major  undertaking.  Therefore,  the  primary  guideline  for 
applying  the  CES  is  NOT  to  apply  it  if  a  course  does  not  have  serious  problems. 
There  are  a  number  of  indicators  that  a  course  may  be  in  trouble.  These  include 
but  are  not  limited  to  the  following: 
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•  Poor  CNET  Level  II  Feedback 

•  High  Attrition 

•  Fleet  Feedback  -  Atlantic  and  Pacific  Training  Assessment  Programs 

•  Student  Comments 

•  Student  Performance  on  Lesson/Module/Course  Tests 

•  Instructor  Review/Comments 

•  Number  of  Setbacks 

If  one  or  more  of  these  indicators  point  to  a  problem,  the  people  responsible  for 
the  course  should  consider  applying  the  CES  to  the  problem  area(s). 


Classifying  Objectives 

In  order  to  use  the  CES,  the  evaluator  must  know  how  to  classify  objectives 
and  how  to  separate  broad  objectives  into  smaller  more  meaningful,  measurable 
and  relevant  objectives.  Background  information  on  these  topics  is  available  in 
NPRDC  SR  79-24,  The  Instructional  Quality  Inventory,  II.  User’s  Manual  Ellis, 

\\  ulfeck,  &  Fredericks,  1979)  and  in  NPRDC  SR  83-2,  Handbook  for  Testing  in 
Navy  Schools  (Ellis  A  \\  ulfeck,  1982)  .  However,  enough  guidance  is  provided  in 
this  section  to  enable  accurate  classification  of  objectives. 

There  are  five  different  types  of  objectives.  The  types  are  based  on  the 
types  of  tasks  performed  on  the  job.  Note  that  the  word  ‘‘task’’  is  used  in  a 
generic  sense.  A  task  can  be  as  broad  or  narrow  as  needed  to  accurately  capture 
the  job  requirements.  In  addition,  a  task  can  involve  remembering  and  reporting 
information  as  well  as  physically  doing  something.  Tasks  are  not  limited  to  the 
narrower  “job,”  “duty,”  and  “task”  definitional  hierarchy  seen  in  some 
instructional  development  models.  The  five  types  of  tasks  are: 

Remember  (R) 

Use-Unaided  Transfer  (UUT) 

Use-Unaided  No-Transfer  (UUN) 

Use- Aided  Transfer  (UAT) 

Use- Aided  No-Transfer  (UAN) 


Classifying  objectives  into  one  of  the  five  types  is  a  three  step  process. 

Step  1:  Determine  if  the  student  is  to  Remember  or  Use  the 
information.  A  student  can  either  Remember  information  or  Use  the 
information  to  do  something.  This  distinction  corresponds  to  the  difference 
between  knowledge  and  application.  The  following  two  objectives  illustrate 
the  Remember-Use  distinction. 

Remember:  The  student  will  draw  the  symbol  for  resistor. 

Use:  The  student  will  set  up  a  Simpson  260-5p  multimeter  for 
measuring  resistance. 
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These  two  objectives  differ  with  respect  to  what  the  student  is  supposed  to 
do.  In  the  first  item,  the  student  has  to  Remember  something.  In  the 
second,  he  has  to  apply  or  Use  his  knowledge.  The  Remember-Use 
distinction  is  usually  a  simple  one.  The  determination  can  usually  be  made 
by  looking  at  the  action  in  the  objective  or  test  item.  Typical  action  verbs 
are  listed  in  Figure  1.  The  ones  on  the  left  usually  indicate  Remember 
tasks,  while  the  ones  on  the  right  usually  indicate  Use  tasks. 


Remember 

Use 

name 

apply 

operate 

state  (from  memoy) 

remove 

repair 

list  (from  memory) 

analyze 

adjust 

recall 

derive 

calibrate 

remember 

demonstrate 

replace 

write  (from  memory) 

evaluate 

assemble 

recognize 

solve 

disassemble 

explain  (from  memoy) 

prove 

calculate 

select 

sort 

troubleshoot 

describe 

maintain 

load 

identify 

compute 

predict 

determine 

unload 

Figure  1.  Action  Verbs 

The  distinction  between  the  verbs  Remember  and  Use  may  be  a  bit  tricky 
when  you  have  an  objective  like  the  following: 

Using  the  manual  NA  01-S3AAA,  list  the  location  and  function  of  the 
fuel  dump  valve. 

At  first  glance  it  might  be  tempting  to  classify  this  a  Remember  objective. 
But  on  closer  examination,  what  the  student  is  actually  required  to  do  is  to 
locate  the  information  in  the  technical  manual  and  record  that  information 
on  a  piece  of  paper.  What  the  student  must  demonstrate  is  the  ability  to 
use  the  technical  manual  and  therefore,  this  is  a  Use  objective. 

Step  2:  If  the  task  is  Use,  determine  if  it  is  Use- Aided  or  Use- 
Unaided.  If  it  is  Use-Unaided  the  student  has  no  aids  except  his  own 
memory.  If  it  is  Use-Aided  the  student  has  a  job  aid  to  perform  the  task. 
This  can  be  determined  by  looking  at  the  “conditions”  part  of  the 
objective.  Anything  that  replaces  the  need  for  memoy  counts  as  an  aid. 

AIDS  include: 

1.  A  list  of  procedure  steps  from  a  technical  manual  or  Maintenance 
Requirements  Card  (MRC). 
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2.  A  formula  for  solving  problems  and  instructions  for  using  the 
formula. 

3.  A  statement  of  a  rule  or  a  set  of  guidelines  for  troubleshooting 
and  repairing  a  piece  of  equipment. 

Normal  tools,  materials,  etc.,  are  NOT  aids. 

Step  3:  If  the  task  is  Use  determine  whether  it  is  Transfer  or  No- 
Transfer. 

A  No-Transfer  objective  requires  the  student  to  perform  a  specific 
procedural  task  the  same  way  every  time.  It  consists  of  an  ordered 
sequence  of  steps  designed  to  accomplish  a  specific  task,  which  needs  to  be 
demonstrated  in  only  one  way.  There  is  no  requirement  that  the  student 
transfer  or  generalize  performance  to  new  situations.  A  good  rule  of  thumb 
is  "if  you’ve  seen  one  you’ve  seen  them  all”  or  “if  you’ve  done  it  right  once 
or  twice  you  can  do  it  right  under  any  circumstance"  you  have  a  No- 
Transfer  objective. 

The  following  are  samples  of  No-Transfer  objectives. 

1.  Tie  a  bowline  knot  within  15  seconds.  (Use-Unaided) 

2.  Destroy  classified  documents  under  routine  conditions  using  the  outline 
in  OPNAMNST  5510.1.  (Use-Aided) 

3.  Weigh  a  C02  fire  extinguisher  and  record  its  weight  in  accordance  with 
the  furnished  MRC.  (Use-Aided) 

Key  verbs  that  you  might  see  in  No-Transfer  objectives  are  listed  below. 


apply 

remove 

operate 

replace 

repair 

assemble 

adjust 

produce 

calibrate 

destroy 

Remember  that  a  No-Transfer  objective  can  be  Use-Unaided  or  Use-Aided. 
The  aid  is  a  list  of  steps  to  be  performed. 

A  Transfer  objective  can  also  contain  a  sequence  of  steps.  However, 
Transfer  objectives  can  be  applied  in  a  variety  of  situations  or  on  a  variety 
of  different  equipments  or  objects.  In  other  words,  they  involve  tasks  and 
jobs  that  are  more  complicated  than  no  transfer  objectives.  Instruction  for 
Transfer  objectives  requires  many  different  examples  and  practice  items  so 
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that  as  many  situations  as  possible  are  covered.  For  Transfer  objectives  it 
is  not  possible  to  teach  every  possible  situation  or  example  in  class.  For 
example,  if  you  wanted  to  teach  5th  graders  three  digit  multiplication,  it 
would  not  be  practical  to  give  them  every  possible  three  digit-problem. 
Instead,  students  are  taught  a  rule  for  dealing  with  the  different  types  of 
problems.  You  would  then  give  them  problems  that  represent  the  different 
types  of  possible  problems  so  that  they  could  practice  using  the  rule. 

Finally,  you  would  test  them  on  new  problems  to  see  if  they  had  learned 
how  to  apply  the  rule.  To  decide  if  you  have  a  TRANSFER  objective  you 
need  to  determine  if  students  will  be  required  to  deal  with  problems  or 
situations  on  the  job  that  may  not  be  covered  in  class. 

The  following  are  Transfer  objectives. 

1.  Given  video  tape  recordings  of  radar  scopes  displaying  jamming,  the 
student  will  classify  the  type  of  jamming  used  for  each  display.  (Use- 
Unaided) 

2.  Given  the  formula  for  capacitive  reactance,  instruction  about  how  to 
apply  it,  and  the  values  of  frequency  and  capacitance  from  a  schematic, 
the  student  will  calculate  capacitive  reactance.  (Use-Aided) 

3.  The  student  will  solve  for  total  power  in  a  DC  parallel  circuit.  (Use- 
Unaided) 

Notice  that  Transfer  objectives  may  be  Use-Aided  or  Use-Unaided.  The  aid 
is  at  least  a  statement  of  the  formula  or  rule  to  be  applied  and  should 
include  guidelines  for  when  and  how  to  apply  it.  Key  verbs  that  you  might 
see  in  a  Transfer  objective  are  given  below. 


solve 

find 

derive 

translate 

prove 

program 

calculate 

add 

troubleshoot 

subtract 

In  summary,  there  are  three  steps  in  classifying  an  objective. 

1.  Determine  if  the  student  is  required  to  Remember  or  Use. 

2.  If  the  task  is  Use,  determine  whether  it  is  Use- Aided  or  Use- 
Unaided. 
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3.  If  the  task  is  Use,  determine  whether  it  is  Transfer  or  No  Transfer. 


Objectives  should  be  classified  before  evaluating  objective-  test  consistency 
and  adequacy  (see  page  xxx).  To  classify  objectives  use  the  form  in  Appendix  B. 
Columns  A  and  B  on  this  form  have  space  for  the  objective  number  and  the  type 
of  objective,  respectively.  Note  that  when  objective-  test  consistency  and 
adequacy  are  evaluated  you  will  need  to  fill  out  a  new  form.  This  is  because 
there  is  often  more  than  one  test  item  associated  with  an  objective  (see  step  8 
page  xxx).  If  it  is  desirable  to  avoid  filling  out  the  form  twice,  the  objective  type 
could  be  written  next  to  each  objective  in  the  curriculum  outline  or  the  list  of  the 
course  objectives. 


Separating  Broad  Objectives  into  Smaller  Unit9 

Objectives  are  generally  classified  and  evaluated  in  the  order  they  appear  in 
the  curriculum  outline.  However,  if  an  objective  contains  multiple  verbs  that 
cover  more  than  one  topic  or  require  more  than  one  action,  it  should  be  separated 
into  more  detailed  objectives  which  contain  only  one  topic  (Remember  objectives) 
or  action  (Use  objectives).  This  will  help  pinpoint  and  correct  instructional  and 
testing  problems.  Three  examples  that  illustrate  the  most  common  types  of 
multiple-verb  objectives  and  how  to  separate  them  are  given  below. 


EXAMPLE  1 

1.  When  the  objective  contains  both  Remember  and  Use  verbs. 

Objective  3.0.  Identify  and  demonstrate  the  proper  handcuffing 
procedures  in  accordance  with  (LAW)  applicable  publications. 

This  objective  is  rewritten  as  twro  objectives  because  it  is  both  a 
Remember  and  Use-Unaided  No-Transfer  objective: 

3.0.1.  Identify  the  proper  handcuffing  procedures  LAW  applicable 
publications. 

3.0.2.  Demonstrate  the  proper  handcuffing  procedures  IAW  applicable 
publications. 

EXAMPLE  2 

2.  When  the  objective  contains  two  or  more  Use  verbs. 

Objective  3.6.  Demonstrate  the  identification,  apprehension, 
handcuffing  and  search  techniques  necessary  to  perform  the  duties  of 
a  Master-At-Arms  IAW  applicable  publications. 

Since  there  are  four  distinct  procedural  tasks,  the  objective  is 
rewritten  as  four  separate  objectives: 


3.6.1.  Demonstrate  the  Identification  Techniques  necessary  to 
perform  the  duties  of  a  Master-At-Arms  in  accordance  with  applicable 
publications. 

3.6.2.  Demonstrate  the  Apprehension  Techniques  .  .  . 

3.6.3.  Demonstrate  the  Handcuffing  Techniques  .  .  . 

3.6.4.  Demonstrate  the  Search  Techniques  .  .  . 


EXAMPLE  3 

3.  When  the  objective  is  a  Remember  objective  with  more  than  one  topic. 

Objective  1.9.  Identify  commonly  abused  drugs  by: 

a.  drug  classification 

b.  psychological  and  physiological  effects 

c.  methods  of  abuse 

d.  street  terminology 

e.  odor  (cannabis  sativa  L-marijuana,  only) 

Since  there  are  five  topics,  the  objective  is  rewritten  as  five  separate 
objectives: 

1.9.1.  Identify  commonly  abused  drugs  by  drug  classification. 

1.9.2.  Identify  commonly  abused  drugs  by  psychological  and 
physiological  effects. 

1.9.3.  Identify  commonly  abused  drugs  by  methods  of  abuse. 

1.9.4.  Identify  commonly  abused  drugs  by  street  terminology. 

1.9.5.  Identify  cannabis  sativa  L  (marijuana)  by  odor. 

Note  that  new  objectives  are  numbered  to  conform  with  the  numbering  system 
used  in  the  curriculum  outline  or  objective  list.  This  is  done  to  ease  tracking  or 
matching  the  original  objectives  with  those  used  in  this  evaluation. 
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Objective  and  Test  Item  Adequacy  &  Consistency  Evaluation 

The  information  in  the  Objective  and  Test  Evaluation  Form  (Appendix  B) 
provides  a  framework  that  will  enable  you  to: 

•  Identify  course  training  level 

•  Classify  all  course  objectives 

•  Check  objective  adequacy/appropriateness 

•  Identify  essential  course  objectives 

•  Match  test  items  to  objectives 

•  Check  test  consistency 

•  Check  test  adequacy/appropriateness 

To  complete  this  form  you  will  need: 

1.  All  objectives  related  to  the  part  of  the  course  under  evaluation. 

2.  All  written  test  items  and  all  performance  checklists/rating  scales  related  to 
those  objectives. 

3.  Assistance  from  SME(s)  if  you  are  not  one  yourself. 


Objective  and  test  evaluation  is  a  12  step  process.  The  steps  are 
summarized  in  “job  aid”  form  in  Figure  2.  Detailed  guidelines  for  performing 
each  step  are  provided  below. 


r 


JOB  AID  FOR 

COMPLETING  OBJECTIVE  AND  TEST  EVALUATION  FORM 


Step 

Step 

Step 

Step 

Step 

Step 

Step 

Step 

Step 

Step 

Step 

Step 


1.  Determine  Overall  Course  Training  Goal. 

2.  Enter  Objective  Number  in  Column  A. 

3.  Enter  Task  Type  in  Column  B. 

4.  If  Task  Type  is  Use  Determine  if  a  Supporting  Objective. 

5.  Determine  if  Objective  is  Essential,  Nice-to-Know,  or 
Unnecessary. 

6.  Identify  Objectives  with  Training  Goal  Different  than  Course 
Training  Goal  Determined  in  Step  1. 

7.  Determine  if  Objective  is  Adequate. 

8.  Enter  Test  Item  Numbers  in  Column  K. 

9.  Determine  if  Test  Items  Matches  the  Objective. 

10.  Determine  if  Test  Item  is  Appropriate. 

11.  Determine  if  Test  Item  is  Adequate. 

12.  For  Transfer  Objectives  Determine  if  there  are  Enough 

Test  Items. 


Figure  2.  Job  Aid 
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Step  1:  Determine  Overall  Course  Training  Goal. 

Think  about  the  overall  training  goal  of  the  course.  What  should  the  graduate 
know  and/or  be  able  to  do  upon  completion  of  the  course?  What  are  the 
requirements  of  the  job?  It  is  essential  to  identify  the  course  training  goal 
because  it  is  the  driving  force  in  the  course.  All  course  objectives  should  be 
directly  related  to  the  training  goal  which  in  turn  should  be  directly  related  to 
the  requirements  of  the  job.  Even  courses  that  are  designed  to  prepare  students 
for  follow-on  training,  such  as  Basic  Electricity  and  Electronics  (BEE),  should  be 
directly  related  to  job  requirements.  If  you  are  evaluating  a  preparatory  type 
course,  treat  it  as  a  portion  of  a  training  program  with  the  training  goal  related 
to  the  ultimate  job  requirements.  In  other  words,  the  overall  training  goal  of  the 
BEE  course  should  relate  to  the  job  requirements  after  the  student  graduates 
from  any  follow-on  “A”  school.  If  you  are  not  an  SME,  you  should  consult  with 
SMEs  assigned  to  the  course  as  manager  and/or  instructors  to  determine  the 
training  goal.  Another  aid  to  determining  the  course  goal  is  the  course  mission 
statement.  Most  Navy  courses  have  a  mission  statement.  Be  careful,  however, 
sometimes  the  mission  statement  will  go  beyond  the  actual  course  goal.  The 
three  most  common  training  goals  are  listed  below.  On  the  form,  enter  the  most 
appropriate  goal  under  the  course  title. 


HEAVILY  SUPERMSED  OJT.  The  student  should  be  generally  familiar  with 
terminology,  technical  documentation  and  duties  required  on  the  job.  The 
student  will  have  had  a  minimal  amount  of  hands-on  experience  for  Use 
objectives.  Upon  completion  of  the  course,  the  student  will  perform  most  of  the 
job  under  close  supervision.  Remember  objectives  may  be  tested  at  the 
recognition  level  (multiple  choice,  true-false,  matching)  or  at  the  recall  level  (fill- 
in,  listing,  short  answer).  This  will  be  dependent  upon  criticality  of  the  objective 
and  cost  and  safety  factors. 


MINIMALLY  SUPERMSED  OJT.  The  student  should  receive  enough  hands-on 
training  so  that  upon  graduation  he  or  she  can  perform  the  job  with  very  little 
supervision.  The  supervisor  will  most  likely  not  guide  the  graduate  through  the 
entire  job  but  will  be  available  to  answer  specific  questions.  The  student  will  be 
able  to  locate  and  use  any  relevant  technical  documentation  if  the  job  is  use- 
aided.  The  student  should  possess  more  specific  knowledge  about  job  procedures, 
technical  documentation  and  terminology  than  at  the  Heavily  Supervised  OJT 
Level.  Therefore,  Remember  objectives  are  tested  at  the  recall  level  (fill-in, 
listing,  short  answer). 


SKILLED  PERFORMER.  The  student  should  receive  enough  hands-on  training 
so  that  upon  graduation  from  the  course,  he  or  she  can  perform  the  task  with  no 
assistance  from  a  supervisor.  The  graduate  is  able  to  supervise/train  others 
performing  the  task. 
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Step  2:  Enter  Objective  Number  In  Column  A.  Enter  only  one  objective 
number  and  then  enter  information  in  columns  B  through  Q  that  corresponds  to 
that  objective  before  entering  the  next  objective  number.  DO  NOT  EVALUATE 
BROAD  TERMINAL  OBJECTIVES  THAT  ARE  ENTIRELY  SUPPORTED  BY 
ENABLING  OBJECTIVES.  Remember  to  break  up  objectives  that  contain  two 
or  more  different  task  types,  two  or  more  procedural  tasks  or  more  than  one 
topic. 

Step  3:  Enter  Task  type  Of  Objective  In  Column  B.  The  task  types  are: 
Remember  (R) 

Use-Unaided  Transfer  (UUT) 

Use-Unaided  No-Transfer  (UUN) 

Use- Aided  Transfer  (UAT) 

Use- Aided  No- Transfer  (UAN) 

Step  4:  If  This  Is  A  Use  Objective,  Determine  If  An  Associated 
Supporting  Objective  Is  Necessary  And  If  Necessary  Is  Present.  In 
Column  C  Enter  Yes  If  Necessary  and  Present,  Enter  No  If  Necessary 
But  Not  Present,  Or  Enter  UN  If  Unnecessary.  If  it  is  a  Use-Unaided 
objective,  a  previous  Remember  objective  may  be  needed.  For  example,  if  the 
task  to  be  performed  is  complicated,  dangerous,  or  involves  transfer  there  should 
be  a  previous  Remember  objective.  However,  if  student  is  required  to  perform  a 
straightforward  uncomplicated  No-Transfer  task,  then  a  previous  Remember 
objective  is  probably  unnecessary.  The  reason  for  this  is  that  the  student’s 
memory  for  how  to  perform  the  task  can  be  inferred  from  observing  performance 
of  the  task  in  a  hands-on  laboratory  exercises.  To  determine  whether  or  not  a 
previous  Remember  objective  is  required  consult  with  your  SME.  If  one  is  not 
required  put  “UN™  in  Column  C. 

If  the  task  is  Use-Aided  a  previous  remember  objective  would  be  required  if 
the  Aid  were  complicated,  contained  unfamiliar  terminology,  or  had  other 
characteristics  that  required  students  to  have  to  remember  information  about  it. 

Step  5:  Review  All  Course  Objectives  And  Determine  The  Following: 

5a:  Is  the  objective  ESSENTIAL  (necessary  for  the  job  the  students  are 
being  trained  for)?  If  the  objective  is  essential,  check  Column  D. 

5b:  Is  the  objective  NICE  TO  KNOW  (motivatic  nal  or  related  information 
that  is  not  essential  for  job  performance)?  If  the  objective  is  nice-to-know, 
check  Column  E. 

5c:  Is  the  objective  UNNECESSARY  (unrelated  and  not  needed  for  job 
performance)?  If  the  objective  is  unnecessary,  check  Column  F. 
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Step  6:  Identify  Objectives  That  Have  A  Training  Goal  Different  From 
The  Overall  Course  Training  Goal.  In  Step  1,  you  identified  the  overall 
training  goal  for  the  course.  Is  the  training  goal  different  for  any  of  the  specific 
essential  objectives.  If  it  is,  enter  the  training  goal  number  (1  -  3)  in  Column  G 
where: 

1  =  Heavily  Supervised  OJT 

2  =  Minimally  Supervised  OJT 

3  =  Skilled  Performer 

Step  7:  Determine  If  The  Objective  Is  Adequate.  The  conditions,  standards 
and  action  must  be  appropriate  for  the  work  to  be  performed  on  the  job  or  in 
later  training.  Keep  in  mind  the  course  training  goal  and  remember  the  job  or 
later  training  requirements  when  determining  if  the  objective  is  appropriate.  If  a 
condition,  standard,  or  action  is  not  contained  in  an  objective,  enter  “N”  (for 
NO)  in  the  appropriate  column.  However,  some  objectives  will  contain  "implicit” 
Conditions  or  Standards,  like  "Given  paper  and  pencil,  ...”  or  "with  100% 
accuracy"  and  these  can  be  evaluated  for  appropriateness. 

7a:  Are  the  CONDITIONS  appropriate?  Enter  “Y"  (for  Yes)  or  “N” 
(for  NO)  in  Column  H.  Are  the  conditions  typical  of  those  found  on 
the  job?  The  conditions  may  not  be  typical  of  the  job  but  still  be 
appropriate  if  they  relate  to  the  course  training  goal.  For  example, 
suppose  the  objective  is  Use-  Aided  but  there  is  no  similar  aid  on  the 
job.  If  the  training  goal  is  Heavily  Supervised  OJT,  it  may  be 
appropriate  to  have  an  aid  while  training  even  though  there  is  no  aid 
on  the  job.  For  Use-Unaided  objectives,  are  there  any  aids  on  the 
job?  If  there  are  job  aids  but  the  students  do  not  have  similar  aids 
during  training,  the  conditions  are  probably  not  appropriate. 

7b:  Are  the  STANDARDS  appropriate  for  the  work  to  be  performed 
on  the  job?  Is  the  time  limit,  accuracy  requirement,  etc.  appropriate 
for  the  job  and  the  course  training  goal?  Enter  “Y”  or  “N”  in  Column 
I. 

7c:  Is  the  ACTION  appropriate  for  the  work  to  be  performed  on  the 
job?  Is  it  the  action  the  student  will  perform  on  the  job?  ENTER 
“Y”  OR  “N"  in  Column  J. 

Step  8:  Enter  The  Test  Item  Numbers  For  All  Items  Associated  With 
The  Objective  In  Column  K.  Obtain  a  copy  of  the  tests  so  they  can  be 
matched  on  the  Objective  and  Test  Review  Form  (Appendix  B).  If  there  are 
multiple  or  alternative  versions  of  the  written  tests  it  is  only  necessary  to  test  one 
version  if  the  tests  involve  only  minor  variations,  such  as  the  rearrangement  of 
test  items  within  the  test  versions.  If  you  have  more  than  one  test  version  that 
contain  different  questions  (not  just  sequenced  differently),  match  the  test 
questions  from  all  test  versions  to  appropriate  objectives.  In  other  words,  treat 
separate  versions  and  daily  quizzes  as  one  big  test  to  be  matched  with  objectives. 


Enter  only  one  test  item  number  on  each  line  in  column  K.  Continue 
entering  vertically  down  the  form  until  you  have  listed  all  test  item  numbers 
after  their  associated  objective.  If  several  test  items  are  found  for  a  given 
objective  will  require  a  separate  line  on  the  form.  For  this  reason  it  is  important 
to  complete  all  the  columns  for  an  objective  before  moving  to  the  next  objective. 
Columns  A  and  K  will  then  match  up  correctly.  For  any  objective  that  is  not 
tested  put  “NT”  in  Column  K. 

For  use  objectives  that  require  hands  on  performance  find  out  if  there  is  an 
associated  Lab  Exercise.  If  there  is  a  lab  enter  LX  in  Column  K  to  indicate  that 
there  is  a  lab  exercise  for  the  objective.  For  courses  at  the  Heavily  Supervised 
OJT  training  level  the  lab  exercise  can  serve  as  both  practice  and  test  item; 
therefore,  do  not  count  a  test  item  as  missing  if  the  lab  exercise  in  Heavily 
Supervised  OJT  course  does  not  look  like  a  formal  test.  If  there  is  no  lab  and  the 
objective  requires  that  there  be  one  put  “NT”  in  Column  K.  Finally,  some  test 
items  may  not  relate  to  a  course  objective.  Enter  those  test  items  in  Column  K 
at  the  end  of  the  checklist  and,  in  Column  A  enter  N.O.  to  indicate  that  there  is 
no  objective  associated  with  the  test  item. 

Step  9:  Determine  Whether  Or  Not  The  Test  Item  Matches  The 
Objective.  Rating  Scales/Performance  Checklists  should  also  be  evaluated  as 
test  items.  If  a  Lab  Exercise  has  no  associated  checklist/rating  scale,  leave 
COLUMNS  L  through  N  blank. 

9a:  Do  the  CONDITIONS  under  which  the  test  item  is  administered  mat^h 
the  conditions  in  the  objective?  Are  the  “Givens”  the  same?  in  the 
objective?  Enter  “V”  or  “N”  in  Column  L. 

9b:  Does  the  STAND. ARD  for  scoring  the  test  item  match  the  standard  in 
the  objective?  Enter  “V”  or  “N”  in  Column  M. 

9c:  Does  the  ACTION  in  the  test  item  match  the  action  in  the  objective? 
Enter  “Y”  or  “N”  in  Column  N. 

Appendix  A  provides  an  example  of  this  process.  For  further  examples  see  the 
Instructional  Quality  Inventory,  II:  Users  Manual  (Ellis,  Wulfeck,  A  Fredericks, 
1979). 

Step  10:  Determine  Whether  The  Test  Item  Is  Appropriate  For  The 
Training  Goal  Of  The  Objective.  Enter  “Y”  Or  “N”  In  Column  O.  In 

Step  8,  you  matched  test  conditions,  standards,  and  actions  to  the  objective.  It 
is  possible  that  the  test  item  and  objective  match  periectly  but  the  test  item  may 
not  be  appropriate  because  the  objective  was  not  appropriate  for  the  job  or 
training  goal.  (Of  course  any  test  item  that  failed  to  match  in  Step  8  is  by 
definition  not  appropriate  and  therefore  gets  a  “N”  in  Column  O.)  For  example, 
suppose  a  test  item  is  multiple  choice  and  Minimally  Supervised  OJT  was 
identified  as  the  training  goal  for  the  course.  The  multiple  choice  test  item  is  not 
appropriate  because  Minimally  Supervised  OJT  requires  constructed  response 
type  test  items  (short  answer,  fill-in,  listing)  for  Remember  objectives  (see  Figure 


3).  But  it  is  important  to  note  that  even  at  the  Heavily  Supervised  OJT  level,  it 
may  not  be  appropriate  to  test  a  Remember  objective  with  a  selected  response 
test  (multiple  choice,  true-false,  matching).  It  may  be  imperative  that  the  student 
memorize  the  material.  This  is  often  the  case  when  personnel  or  equipment 
safety  is  at  stake.  In  those  cases,  you  want  to  make  certain  that  the  student  can 
recall  the  information  and  not  merely  recognize  the  correct  answer.  Consult 
with  an  SME  to  determine  the  criticality  of  the  objective  before  deciding  if  the 
test  item  is  appropriate.  In  general,  standards  should  become  harder  and 
conditions  more  typical  of  the  job,  as  you  move  to  a  higher  training  goal.  For 
example,  for  performance  tests  there  should  be  longer  time  limits,  more  errors 
allowed,  and  more  aids  for  Heavily  Supervised  OJT  than  for  Minimally 
Supervised  OJT  and  Skilled  Performer.  For  Skilled  Performer,  the  student  may 
be  required  to  perform  the  task  several  times  without  any  errors  and  without  the 
assistance  of  a  training  aid. 

If  the  objective  is  identified  as  unnecessary  in  Step  5,  any  associated  test 
items  should  be  considered  inappropriate.  An  unnecessary  objective  should  not 
be  included  in  the  course.  For  an  objective  that  is  nice-to-know,  associated  test 
items  should  be  considered  inappropriate  if  the  objective  is  included  in  the  course 
only  to  motivate  students.  However,  if  an  objective  is  nice-to-know  because  it 
relates  to  some  future  job  requirements  (as  opposed  to  immediate  requirements), 
associated  test  items  may  be  considered  appropriate. 

If  there  is  a  lab  exercise  (LX  entered  in  Column  K)  with  no  associated  rating 
scale/checklist,  you  can  not  evaluate  the  appropriateness  of  the  lab  exercise. 
Leave  Column  O  blank.  Figure  3  on  the  following  page  should  help  determine 
test  item  appropriateness. 


TRAINING  GOAL 

TEST  ITEM  TYPE 

Heavily  Supervised 

Perform  only  once  with 
several  errors  allowed.  If 
aided,  aid  must  be 
provided.  Remember 
objectives  may  be  tested 
with  either  constructed  or 
selected  response  items 
based  on  SME  criticality 
rating. 

Minimally  Supervised 

Perform  task  a  few  times 
with  fewer  errors  allowed. 

If  aided,  aid  must  be 
provided.  Remember 
objectives  must  be  tested 
with  constructed  response 
items. 

Skilled  Performer 

Perform  task  several  times 
with  no  errors  allowed. 

Same  as  minimally 
supervised  for  aided  and 
unaided  objectives. 

Figure  3.  Test  Item  Format  Determination 

Step  11:  Determine  Whether  The  Test  Item  Is  Adequate.  Enter  Yes  Or 
No  In  Column  P.  (If  the  item  failed  to  match  the  objective  in  Step  8  it  is  not 
adequate  and  "N"  should  be  put  in  Column  P.)  Use  the  following  criteria: 

11a:  Is  the  test  item  CLEAR? 

lib:  Is  the  test  item  UNAMBIGUOUS? 

11c:  Is  the  test  item  WELL  CONSTRUCTED? 
lid:  Is  the  test  item  FREE  OF  HINTS? 

lie:  Does  the  test  item  allow  for  COMMON  ERRORS  to  be  made? 

Ilf:  For  L^se  tasks,  is  the  Rating  Scale/Checklist  w-ell  constructed?  Are 
there  enough  items  to  test  the  performance?  Are  the  instructions  clear?  If 
there  is  no  associated  Rating  Scale/Checklist  for  a  Lab  Exercise  (LX),  enter 
“N"  in  Column  P. 

For  specific  details  on  how  to  apply  these  criteria  along  with  explained  examples 
see  the  Handbook  for  Testing  in  Navy  Schools  (NPRDC  Spec.  Rep.  83-2)  Navy 
Personnel  Research  and  Development  Center,  San  Diego,  (Ellis  &  Wulfeck,  1982). 

Step  12:  For  Use-Unaided  and  Use- Aided  Transfer  Objectives, 
Determine  If  There  Are  Enough  Test  Items.  Enter  “Y”  Or  “N”  In 
Column  Q.  If  the  objective  is  not  a  Transfer  objective  enter  N/A  in  Column  Q. 
Developing  tests  for  objectives  that  require  transfer  is  different  than  developing 
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tests  for  objectives  that  do  not.  A  No-Transfer  objective  defines  only  one  or  a 
small  number  of  possible  test  items,  and  these  items  are  ’’seen  before"  during 
training.  For  example,  if  an  objective  requires  the  student  to  recall  the  names  of 
the  parts  of  a  piece  of  equipment,  the  only  way  to  test  this  is  to  ask  the  student 
to  recall  the  names  of  the  parts,  and  the  student  will  have  practiced  recalling 
these  names  during  training.  Similarly,  if  an  objective  requires  the  student  to 
perform  a  procedure  on  a  piece  of  equipment,  the  test  will  require  the  student  to 
perform  the  procedure,  and  the  student  will  have  practiced  this  performance 
during  training. 

A  Transfer  objective  defines  a  large  (or  even  infinite)  number  of  test  items. 
Only  some  of  these  can  be  practiced  during  training,  and  only  some  can  be  given 
on  tests.  For  example,  if  an  objective  requires  the  student  to  use  a  rule  that 
applies  to  a  large  or  infinite  number  of  problems,  some  of  these  will  be  used  as 
examples  or  practice  items  during  instruction,  ana  others  will  appear  on  tests. 
There  will  still  be  many  problems  that  could  have  been  tested  and  that  the 
student  is  supposed  to  be  able  to  solve.  Since  not  all  test  items  from  this  large  or 
even  infinite  number  can  be  administered,  you  need  to  decide  how  many  test 
items  you  should  give  to  the  student.  The  student  should  be  given  test  items 
that  represent  the  various  types  of  problems  and  there  should  be  enough  test 
items  so  that  the  score  the  student  gets  on  the  test  accurately  represents  the 
score  the  student  would  get  on  a  test  containing  all  the  possible  test  items  (if  it 
were  possible  to  construct  such  a  test). 

Again,  for  more  information  on  how  to  accomplish  this  step  and  testing  in 
general,  see  the  Handbook  For  Testing  in  Navy  Schools,  (Ellis  &  Wulfeck,  1982). 
Finally,  see  Appendix  A  for  an  example  of  a  completed  Objective  and  Test 
Evaluation  Form. 
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Presentation  Evaluation 


This  evaluation  is  conducted  while  observing  the  instructor  in  the  classroom 
and  uses  the  forms  in  Appendices  C,  D,  and  E.  Determine  if  you  want  to  observe 
the  entire  course  or  a  segment  of  the  course.  Identify  and  classify  the  objectives 
associated  with  the  segment(s)  prior  to  the  observation.  Remember  to  separate 
broad  objectives  into  smaller  units.  If  the  objective  and  test  evaluation  has  been 
conducted,  the  objective  numbers  should  match  the  numbers  assigned  on  that 
form  (Appendix  B).  Obtain  copies  of  all  related  instructor  and  student  guides  as 
you  will  evaluate  the  presentation  and  the  guides  together. 

The  three  forms  completed  during  the  presentation  evaluation  are: 

1)  Presentation  Consistency  Form,  Appendix  C 

2)  Presentation  Adequacy  Form,  Appendix  D 

3)  Instructional  Effectiveness  Checklist,  Appendix  E 

If  the  presentation  is  consistent  it  will  contain  the  right  combination  of 
presentation  parts  or  components  that  are  required  for  that  particular  type  of 
objective.  If  the  presentation  is  adequate  it  will  incorporate  a  number  of 
instructional  design  principles  which  promote  student  learning.  The  Instructional 
Effectiveness  Checklist  contains  a  series  of  yes/no  questions  that  relate  to 
presentation  effectiveness. 

Presentation  Consistency  (Appendix  C) 

For  a  presentation  to  be  consistent  with  an  objective,  it  must  teach  to  the  task 
type  of  that  objective  and  contain  the  right  combination  of  presentation 
components.  That  is,  different  types  of  presentation  components  are  required  for 
different  types  of  tasks.  The  five  main  presentation  components  are: 

1.  STATEMENT.  Presents  essential  information  needed  by  student  to  master 
the  objective. 

FOR  EXAMPLE,  a  statement  for  a  Remember  objective  that  requires  the 
student  to  know  the  part  names  of  a  lathe  could  involve  the  instructor 
naming  the  parts  on  a  diagram  or  the  actual  equipment  in  front  of  the 
class.  It  could  also  be  a  labeled  diagram  in  the  student  guide  that  the 
instructor  tells  the  student  to  study.  For  an  objective  that  requires  the 
student  to  know  the  steps  for  preparing  an  unclassified  message  for 
transmittal,  the  instructor  could  orally  state  or  visually  present  (on  a 
chalkboard,  viewgraph,  or  slide,  for  example)  each  step  to  the  class. 

2.  PRACTICE  REMEMBERING.  Provides  an  opportunity  for  the  student  to 
test  her  or  his  ability  by  recalling  or  recognizing  specific  information  presented  in 
the  statement. 
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FOR  EXAMPLE,  the  instructor  could  orally  quiz  students  about  the  parts 
of  a  lathe  or  the  steps  for  transmitting  a  message  during  the  presentation, 
or  could  administer  written  practice  during  class,  or  could  refer  students  to 
practice  questions  in  the  student  guide  to  be  done  as  homework. 

3.  EXAMPLE.  A  demonstration  of  the  task  is  performed. 

FOR  EXAMPLE,  the  instructor  could  physically  demonstrate  how  to 
operate  a  lathe  or  how  to  transmit  a  message  in  front  of  the  students  either 
in  the  classroom  or  the  laboratory,  or  students  could  view  slides,  a  movie,  a 
videotape,  a  videodisc,  etc.  showing  how  to  do  the  task. 

4.  PRACTICE  L’SING.  The  student  is  given  an  opportunity  to  perform  the 
task. 

FOR  EXAMPLE,  the  student  could  be  given  a  project  to  complete  during 
which  he  or  she  would  operate  a  lathe  or  transmit  a  message.  For  tasks  in 
which  cost  and  safety  need  to  be  considered  simulators  could/should  be 
used  to  provide  practice  using. 

5.  FEEDBACK.  For  practice  remembering  the  student  should  be  given  feedback 
about  the  correctness  of  his  or  her  answer.  For  practice  using  the  student  should 
be  given  feedback  about  the  quality  of  his  or  her  performance. 

FOR  EXAMPLE,  for  practice  remembering  information  such  as  lathe  part 
names  or  steps  for  message  transmittal  that  occurs  in  the  classroom  or  is 
done  as  homework,  the  instructor  should  review  each  question  and  provide 
the  correct  answer.  For  practice  using  in  the  classroom  or  laboratory  the 
instructor  should  either  give  oral  feedback  or  a  written  evaluation  to  the 
student. 

Again,  different  combinations  of  these  components  are  required  depending  on 
the  task  type  of  the  objective.  Figure  4  shows  which  components  are  required 
for  each  task  type. 
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Required  Presentation  Components 

Task  Type 

Statement 

Practice 

Remembering 

w/Feedback 

Examples 

Practice 

Using 

w/Feedback 

Remember 

Required 

Required 

Not  Required 

Not  Required 

Use-Unaided 
No  Transfer 

Required  if 
no  recent 

remember 

objective 

Required  if 
no  recent 

remember 

objective 

Required 

Required 

Use  Aided 

No  Transfer 

Aid  replaces 
statement 

Not  Required 

Required 
with  aid 

Required 
with  aid 

Use-Unaided 

Transfer 

Required  if 
no  recent 

remem  her 
objective 

Required  if 
no  recent 

remember 

objective 

Required  with 
variety  of 
examples 

Required  with 
variety  of 
practice 
problems 

Use- Aided 
Transfer 

Aid  replaces 
Statement 

Not  Required 

Required  with 
variety  of 
examples  and 
aid 

Required  with 
variety  of 
problems  and 
aid 

Figure  4.  Presentation  Components 


The  Presentation  Consistency  Form,  Appendix  C,  is  designed  to  record  the 
presence  and  completeness  of  required  presentation  components.  When 
considering  whether  a  component  is  present  and  complete,  evaluate  the 
instructor's  presentation  in  the  classroom  in  conjunction  with  the  instructor  and 
student  guides.  The  instructor  guide  is  evaluated  to  check  (1)  if  it  is  consistent 
with  the  objectives.  (2)  if  it  is  consistent,  is  the  instructor  using  it  correctly,  and 
(3)  if  it  is  not  consistent,  is  the  instructor  correcting  the  deficiency  in  his/her 
presentation.  The  student  guide  is  evaluated  because  it  frequently  contains 
practice  exercises  that  students  are  required  to  do  as  homework.  These  exercises 
need  to  be  evaluated  to  see  if  they  meet  the  “practice  and  feedback  component” 
requirements.  If  they  do,  mark  practice  remembering  and  feedback  as  present 
and  complete.  Also,  for  courses  at  the  Heavily  Supervised  OJT  training  level  a 
lab  exercise  can  serve  as  both  practice  and  a  test.  In  this  instance,  do  not  count 
practice  as  missing  if  the  lab  exercise  is  only  done  once. 
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Prior  to  classroom  observation,  complete  the  following  steps: 

1.  Enter  objective  numbers  on  the  first  line  of  the  Presentation  Consistency 
Form  (Appendix  C).  (Eight  objectives  may  be  evaluated  on  each  form). 

The  objective  numbers  should  match  the  numbers  assigned  on  the 
Objective  k  Test  Evaluation  Form  (Appendix  B).  Remember  that 
objectives  with  multiple  verbs  may  need  to  be  rewritten  so  that  they 
contain  only  one  topic  (Remember  objectives)  or  action  (Use  objectives). 

2.  Determine  the  task  type  of  each  objective. 

3.  For  each  objective,  put  an  “X”  in  the  presentation  component  blocks 
that  are  not  required  for  the  objective  task  type. 

During  classroom  observation  or  during  instructor  and  student  guide 
evaluation,  indicate  the  presence  and  completeness  of  each  presentation 
component  by  writing  in  one  of  the  following  letters  in  each  block: 

0  =  component  COMPLETE 
1  =  component  INCOMPLETE 
N  =  component  NOT  PRESENT 

See  Appendix  A  for  an  example  of  a  completed  Presentation  Consistency  form  or 
Appendix  C  for  a  blank  copy  of  the  form. 

Presentation  Adequacy  (Appendix  D) 

The  presentation  may  include  the  right  combination  of  presentation 
components  and  still  not  communicate  to  the  student  because  (l)  the  contents 
are  jumbled  together  in  such  a  way  that  it  is  difficult  to  distinguish  one 
component  from  another  or  (2)  the  subject  matter  requires  supporting 
information  or  additional  explanation  to  answer  student  questions.  Presentation 
Adequacy  concerns  the  organization  and  format  of  materials  and  the 
communication  of  supporting  information  where  needed  by  the  student. 

The  Presentation  Adequacy  Form  (Appendix  D)is  designed  to  record  the 
adequacy  of  each  presentation  component  with  regard  to  the  following  criteria: 

1.  SEPARATED  (Sep).  Is  the  component  separated  from  the  rest  of  instruction? 

FOR  EXAMPLE,  does  the  instructor  pause  before  introducing  the  new 
component,  put  it  on  the  blackboard,  or  display  it  with  a  viewrgraph  or 
slide? 

2.  IDENTIFIED  (Ider.).  Is  the  component  identified  so  students  know  what  it  is? 

FOR  EXAMPLE,  does  the  instructor  state  orally  that  he  or  she  is  giving  an 
example  or  a  definition?  If  the  practice  is  in  the  student  guide,  is  it  labeled? 


-  20  - 


3.  CLEARLY  STATED  (Clear).  Is  the  component  clearly  stated  so  that  the 
student  can  understand  it? 

FOR  EXAMPLE,  can  students  understand  the  terminology  used?  Is  the 
presentation  unambiguous  and  not  confusing? 

4.  HELPS.  Does  the  component  contain  something  to  help  the  student  better 
understand? 

FOR  EXAMPLE,  is  there  a  memory  aid,  demonstration  of  each  step, 
highlight  of  critical  characteristics,  etc?  Does  the  instructor  explain  new 
information  by  relating  it  to  something  the  student  already  knows? 

5.  JOB  ORIENTED  (Job).  Is  component  oriented  toward  job  performance? 

THAT  IS,  is  the  student  told  how  what  is  being  taught  is  required  for 
competent  job  performance?  Is  the  component  presented  in  the  context  of 
the  job’ 

6.  ENOlrGH.  Is  there  enough  practice  or  enough  examples? 

FOR  EXAMPLE,  for  Transfer  tasks  there  need  to  be  enough  practice  items 
and  examples  so  that  students  can  perform  in  situations  that  they  have  not 
encountered  during  training.  For  NoTransfer  tasks  they  need  to  have 
enough  practice  to  perform  at  the  course  training  goal.  (See  the  Handbook 
for  Testing  in  Navy  Schools,  Ellis  <Sr  Wulfeck,  1982,  for  more  details  and 
examples.) 

7.  COMMON  ERRORS  (Errors).  Do  practice  and  feedback  allow  for  errors 
commonly  made  on  the  job?  Do  examples  show  why  common  errors  are  wrong? 

FOR  EXAMPLE,  on  the  job  are  safety  precautions  easily  ignored,  or  are 
important  steps  frequently  left  out? 

8.  EASY  TO  HARD  (E-H).  Is  practice  easy  to  hard  when  appropriate?  Are 
early  examples  simplified  for  complex  tasks? 

FOR  EXAMPLE,  if  the  task  was  to  field  strip  an  M-16  rifle  in  the  dark  it 
would  not  be  appropriate  to  begin  training  in  the  dark. 

9.  STUDENT  INVOLVEMENT  (Invl).  Does  the  instructor  encourage  student 
involvement? 
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FOR  EXAMPLE,  during  the  presentation,  does  the  instructor  ask  the 
students  to  supply  the  definition,  steps  of  procedure,  etc  from  the  student 
guide  or  from  memory?  During  a  demonstration,  does  the  instructor  call  on 
students  to  give  steps  of  procedure,  predictions,  etc?  Does  the  instructor 
pass  around  a  piece  of  equipment  during  the  lecture  or  demonstration? 

Note  that  not  all  criteria  apply  to  each  component  and  therefore  certain  blocks 
on  the  form  have  been  X’d  out.  See  Appendix  A  for  an  example  of  a  completed 
Presentation  Adequacy  Form. 

Prior  to  classroom  observation  complete  the  following  steps: 

1.  Enter  the  objective  number  on  each  form.  Only  one  objective  is 
evaluated  on  each  form.  The  objective  numbers  should  match  the  numbers 
used  on  the  Presentation  Consistency  Form  (Appendix  C). 

2.  Put  an  “X”in  the  boxes  for  all  presentation  components  not  required  by 
objective  task  type.  Refer  back  to  the  Presentation  Consistency  Form  and 
X  out  the  same  presentation  components. 

During  classroom  observation,  indicate  whether  each  criterion  is  met  by 
writing  in  one  of  the  following  letters  in  each  block: 

Y  =  criterion  met 
N  =  criterion  not  met 

NA  =  Not  appropriate  for  particular  type  of  task 

Remember  that  some  components  may  be  contained  in  the  student  guide  and 
should  also  be  rated  for  adequacy. 


Instructional  Effectiveness  Checklist  (Appendix  E) 

The  Instructional  Effectiveness  Checklist  (Appendix  E)  is  designed  to  record 
classroom  activities  and  conditions  that  affect  student  learning  and  motivation. 
The  form  is  straight-forward;  simply  check  the  appropriate  column  or  record  the 
requested  information  in  the  identified  block. 
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Final  Word 


Since  applying  the  CES  requires  a  lot  of  work,  the  CES  should  be  used  only 
on  courses  or  portions  of  courses  with  serious  problems.  If  there  are  problems, 
the  CES  is  extremely  effective  in  detecting  and  correcting  deficiencies.  In  a 
recent  application  of  a  prototype  of  the  CES,  academic  attrition  was  reduced  by 
over  50  percent  (Ellis,  1985).  The  CES  is  specifically  designed  to  detect  the 
following: 

Inconsistencies  among  objectives,  test  items,  and  presentation  (which 
includes  classroom  presentation  and  the  instructor  and  student  guides); 

Inadequate  objectives,  test  items,  and  presentation; 

Ineffective  presentation  including  laboratory  exercises; 

Objectives  and  test  items  that  are  not  consistent  with  the  overall  course 
goals; 

Objectives  that  are  not  tested; 

Test  items  that  are  not  associated  with  objectives; 

Objectives  (and  any  associated  test  items)  that  are  nice-to-know  or 
unnecessary. 

If  in  reviewing  your  course,  module,  or  lesson  you  find  any  of  these  problems 
you  should  work  with  the  course  SMEs  (if  you  are  not  one  yourself)  to  correct 
them.  Remember,  the  primary  goal  of  the  CES  is  to  ensure  that  classroom 
instruction  is  adequately  presented  and  is  consistent  with  the  requirements  of  the 
job  the  students  will  be  performing  when  they  graduate  from  the  course. 


Recommendations 

The  Chief  of  Naval  Education  and  Training  is  currently  developing  a  Navy 
wide  policy  for  overall  quality  control  of  training  and  training  management.  It  is 
recommended  that  the  Course  Evaluation  System  be  incorporated  in  this  policy. 
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Appendix  A 


THE  COURSE  EVALUATION  SYSTEM:  A  WORKED  EXAMPLE 

This  appendix  provides  a  worked  example  of  a  course  evaluation  using  the 
Course  Evaluation  System  (CES).  The  course  used  in  this  example  is  the  Master- 
At-Arms  (Afloat)  Indoctrination,  K-830-2122,  which  was  taught  at  the  Fleet 
Training  Center,  San  Diego,  CA.  This  course  is  intended  to  provide  basic  Naval 
Law  Enforcement  Training  for  officers  and  petty  officers.  The  students  are 
taught  the  technical  knowledge  and  physical  skills  required  to  perform  basic 
Master-at-Arms  duties.  The  course  lasts  5  days  and  consists  of  both  classroom 
and  laboratory  instruction. 

The  portion  of  the  course  evaluated  in  this  example  is  Lesson  Topic  3.1  -  Use 
and  Application  of  the  Police  Baton.  The  topic  consists  of  three  enabling 
objectives  that  require  0.5  hours  of  classroom  presentation  and  1.5  hours  of 
laboratory  exercise. 

This  example  is  divided  into  four  sections.  The  first  three  sections  are 
Objective  Classification  (including  separating  broad  objectives  into  smaller  units). 
Objective  and  Test  Adequacy  and  Consistency  Evaluation,  and  Presentation 
Evaluation.  These  sections  are  based  on  the  the  three  major  evaluation  tasks  of 
the  CES.  The  fourth  and  final  section  lists  the  course  revisions  that  need  to  be 
made  based  on  the  CES  findings.  Figures  1  through  6  (beginning  on  page  A-8) 
are  the  completed  CES  forms  for  this  course. 


Objective  Classification 

Figure  1  is  the  lesson  topic  page  taken  from  the  course  curriculum  outline. 
Note  that  it  shows  terminal  objective  3.0  which  is  partially  support  by  this  lesson 
topic  and  the  three  enabling  objectives  which  are  supported  entirely  by  this 
lesson  topic.  The  first  thing  to  do  in  the  CES  is  to  classify  the  enabling 
objectives. 


Objective  3.1. 

Step  1.  Is  the  student  required  to  Remember  or  Use?  The  verb  in  this  objective 
is  “demonstrate”  which  is  a  Use  action  verb.  (See  CES  page  2). 

Step  2.  If  Use,  is  it  Use-Unaided  or  Use- Aided.  The  objective  does  not  indicate 
that  the  student  will  be  given  diagrams  or  instructions  to  look  at  while 
demonstrating  the  movements;  therefore  it  is  Use-Unaided.  (See  CES  page  3). 

Step  3.  Does  the  objective  require  Transfer  or  No-Transfer.  The  phrase  “basic 
movements  of  the  Koga  Method  of  Police  Baton  Utilization”  indicates  that  this 
task  is  procedural  and  is  performed  in  a  standard  manner.  Therefore,  this 
objective  is  Use-Unaided  No-Transfer  (UUN).  (See  CES  pages  3-5). 
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Objective  3.2 

Step  1.  The  verb  in  this  objective  is  also  “demonstrate,”  so  it  is  a  Use  objective. 
Step  2.  Again,  no  aids  are  indicated,  so  it  is  Use-Unaided. 

Step  3.  Again,  this  is  a  procedural  task  performed  in  a  standard  manner,  so  the 
objective  is  Use-Unaided  No  Transfer  (UUN). 


Objective  3.3 

Step  1.  The  verb  in  this  objective  is  “identify,”  which  is  a  Remember  verb. 
Therefore,  this  objective  is  Remember  (R)  and  Steps  2  and  3  do  not  have  to  be 
done. 

Note  that  none  of  these  objectives  involve  multiple  actions  or  topics  and, 
therefore,  do  not  have  to  separated  into  smaller  units. 


Objective  and  Test  Adequacy  &  Consistency  Evaluation 

For  this  evaluation,  you  need  to  get  the  test  items  that  go  with  the 
objectives.  You  also  need  the  Objective  and  Test  Review  form,  which  is 
Appendix  B.  Finally,  if  you  are  not  a  subject  matter  expert  (SME),  you  need  to 
have  one  (preferrablv  two  or  three)  available  as  you  complete  the  form.  (See  CES 
page  8  for  additional  explanation). 

The  test  items  you  need  (taken  from  the  course  final  exam)  are  shown  in 
Figure  2.  Figure  3  is  a  completed  Objective  and  Test  Review  form  for  objectives 
3.1,  3.2.  and  3.3.  To  complete  this  form,  we  consulted  with  the  course  manager 
and  two  of  the  course  instructors.  They  provided  the  necessary  subject  matter 
expertise.  The  methods  we  used  for  completing  each  column  of  the  form  are 
discussed  below  . 


Step  1.  Determine  Training  Goal. 

For  this  step  we  consulted  the  course  manager  and  an  instructor  and  found  that 
the  training  goal  was  Heavily  Supervised  On  the  Job  Training.  We  entered  this 
information  on  the  line  provided  below  the  course  title  on  the  left  hand  side  of 
the  form.  (See  Figure  3). 


Step  2.  Enter  Objective  Number  in  Column  A. 

This  is  routine.  We  copied  the  objective  numbers  from  the  curriculum  outline  to 
the  form  in  Column  A.  (See  Figure  3). 


Step  3.  Enter  Task  Type  in  Column  B. 

For  each  objective,  we  entered  in  Column  B  the  task  type  that  we  determined  in 
the  Objective  Classification  phase. 
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Step  4.  If  Use  Objective,  is  there  a  Supporting  Objective. 

Objectives  3.1  and  3.2  are  Use  objectives  and  there  is  no  supporting  Remember 
objectives.  In  this  case,  however,  this  is  not  really  a  problem  because  the 
training  goal  is  heavily  supervised  OJT  and  the  task  is  not  complicated  or 
dangerous.  Therefore,  knowledge  of  how  to  perform  the  task  can  be  evaluated  by 
observing  the  students  perform  it.  so  you  don't  need  a  separate  Remember 
objective.  We  put  “UN”  (for  unnecessary)  in  Column  C  (Figure  3). 


Step  5.  Determine  if  Objectives  are  Essential,  Nice-To-Know,  or 
Unnecessary. 

We  consulted  our  SMEs  who  told  us  that  all  three  objectives  were  essential  for 
job  performance.  Therefore,  we  put  a  check  in  Column  D  (Figure  3)  for  each 
objective. 


Step  6.  Identify  Objectives  with  Different  Training  Goals. 

According  to  our  SMEs.  none  of  the  three  enabling  objectives  had  a  different 
training  goal.  so.  we  left  Column  G  (Figure  3)  blank. 


Step  7.  Determine  if  the  Objective  is  Adequate. 

We  asked  our  SMEs  if  the  conditions,  standards,  and  action  for  each  of  the 
objectives  were  appropriate  for  the  job  and  they  told  us  that  everything  was 
appropriate  for  objectives  3.1  and  3.2  so.  for  these  objectives,  we  put  Yes  in 
Columns  H,  I,  and  J  (Figure  3). 

Objective  3.3  did  not  have  appropriate  conditions,  action,  and  standards  because 
the  objective  involved  knowing  information  critical  for  personal  safety;  that  is 
where  not  to  strike  on  the  body  with  a  baton.  Therefore,  we  put  No  in  Columns 
H.  I  and  J  (Figure  3). 


Step  8.  Enter  Test  Item  Number. 

For  objectives  3.1  and  3.2  we  entered  LX  in  Column  K  (Figure  3)  because  the 
curriculum  outline  and  the  SMEs  indicated  that  there  was  a  laboratory'  exercise 
(LX)  for  these  objectives.  For  objective  3.3,  we  identifed  items  46  and  47  from 
Figure  2  as  possibly  testing  the  objective.  We  entered  these  numbers  on  separate 
lines  in  two  different  boxes  in  Column  K  (Figure  3). 


Step  9.  Determine  if  Test  Items  Match  the  Objective 

For  objectives  3.1  and  3.2,  we  had  to  leave  Columns  L  through  X  (Figure  3) 
blank  because  there  were  no  checklists/rating  scales  for  the  lab  exercises.  The 
evaluation  of  student  performance  was  left  up  to  the  instructor’s  personal 
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judgement.  This  situation  can  lead  to  serious  problems.  For  example,  different 
instructors  may  evaluate  the  same  student  differently  because  they  have  different 
ideas  about  what  is  correct.  In  addition,  a  new  instructor  may  be  confused  or 
have  insufficient  knowledge  to  accurately  evaluate  performance.  We 
recommended  that  written  checklists/rating  scales  be  developed  for  objectives  3.1 
and  3.2. 

For  objective  3.3,  we  tried  to  match  the  conditions,  standards,  and  actions  in  test 
items  46  and  47  (see  Figure  2)  with  those  in  the  objective.  For  item  46,  the 
conditions  match  (implied  paper  and  pencil)  and  the  standard  matches  (70 
percent  to  pass  the  final  exam),  however;  the  action  in  the  objective,  “identify 
the  non  striking  areas,"  does  not  match  the  action  in  item  46,  which  is  to 
identify  what  “you  may  remove  from  the  suspect."  Therefore,  we  put  a  Y  in 
Columns  L  and  M  and  an  N  in  Column  N. 

For  item  47.  the  conditions,  action,  and  standard  all  match  because  the  subject  is 
asked  to  identify  “the  three  areas  of  the  human  body  the  Master- at-Arms  will 
not  strike."  So.  we  put  a  Y  in  Columns  L,  M,  and  N. 


Step  10.  Does  Test  Item  Match  Training  Goal? 

For  the  lab  exercises  associated  with  objective  3.1  and  3.2.  we  had  to  leave 
Column  O  blank  because  there  were  no  written  checklists/rating  scales.  Item  46 
did  not  match  objective  3.3,  therefore  it  can  not  be  appropriate,  so  we  put  “no” 
in  Column  O.  Item  47  did  match  and,  because  the  course  is  Heavily  Supervised 
OJT  multiple  choice  items  may  be  appropriate,  however,  in  this  case  we  checked 
with  our  SME  because  the  item  seemed  to  involve  health  and  safety  (that  is, 
knowing  which  areas  of  the  body  not  to  strike).  Our  SME  told  us  that  the 
information  covered  by  the  item  was  critical  and  should  be  memorized,  so 
multiple  choice  is  not  appropriate  and  we  put  an  "N"  in  Column  O  for  item  47. 


Step  11.  Is  the  Test  Item  Adequate? 

Using  the  criteria  in  the  CES  we  determined  that  the  test  item  47  was  not 
adequate,  and  we  put  N  in  Column  P.  There  were  no  written  checklists/rating 
scales  for  objectives  3.1  and  3.2,  and  item  46  did  not  match,  so  for  these  items  we 
put  N  in  Column  P. 


Step  12.  For  Transfer  Objectives  Are  There  Enough  Test  Items? 

None  of  the  objectives  reviewed  were  transfer  objective  so  we  put  N/A  in 
Column  Q. 


A-4 


Presentation  Evaluation 

In  this  section,  the  presentation  will  be  evaluated  for  consistency  with  the 
objectives  and  test  items  and  for  adequacy.  The  instructor  and  the  classroom 
environment  will  also  be  evaluated. 


Presentation  Consistency 

Presentation  consistency  is  evaluated  using  the  Presentation  Consistency 
Form  from  Appendix  C.  To  complete  the  form  you  need  to  collect  copies  of  the 
instructor  and  student  guides  associated  with  the  objectives,  and  then  you  need 
to  sit  in  on  the  instructional  presentation,  both  lecture  and  lab.  The  first  step  is 
to  enter  the  objective  numbers  on  the  first  line  of  the  form  (Figure  4,  there  is 
room  for  eight  objectives).  The  next  step  is  to  determine  what  instructional 
components  are  required  based  on  the  Task  Type  of  each  objective.  Then  the 
components  that  are  not  required  are  Xed  out  on  the  form.  In  the  present 
example  the  Task  Type  for  objectives  3.1  and  3.2  is  Use-Unaided  No-Transfer, 
therefore  all  four  presentation  component^  are  required.  The  task  type  for 
objective  3.3  is  Remember,  so  only  a  statement  and  practice  remembering  with 
feedback  are  required.  We  Xed  out  the  remaining  components  on  the  form  (see 
Figure  4). 

After  reviewing  the  instructor  and  student  guides  and  observing  the 
presentation  (including  the  laboratory),  we  found  that  all  the  components  for  all 
three  objectives  were  present  and  complete  with  the  exception  of  the  feedback  for 
objectives  3.1  and  3.2.  There  was  very  little  time  devoted  to  feedback  about 
performance  in  the  laboratory  exercise.  Note  that  for  objectives  3.1  and  3.2  we 
assumed  that  practice  remembering  was  tested  by  observing  students  perform  the 
procedure  during  the  lab  exercise.  On  the  Presentation  Consistency  Form 
(Figure  4)  we  put  C  in  all  boxes  except  the  feedback  boxes  for  objective  3.1  and 
3.2,  in  these  boxes  we  put  I  See  Figure  4  for  the  completed  form. 

Presentation  Adequacy 

To  evaluate  presentation  adequacy,  you  need  to  get  the  Presentation 
Adequacy  Form  (Appendix  D),  the  student  guide,  and  observe  the  instructional 
presentation,  both  lecture  and  lab.  First,  we  Xed  out  the  boxes  on  the 
Presentation  Adequacy  Form  (Figure  5)  that  are  not  required  for  each  of  the 
three  objectives.  Then  we  evaluated  each  objective  on  the  applicable  adequacy 
criteria  and  put  Ys  and  Ns  in  the  appropriate  boxes  on  the  Presentation 
Adequacy  Form  (see  Figure  5).  Figure  5  shows  that  for  all  three  objectives  the 
instructor  (and/or  the  student  guide-a  part  of  the  overall  presentation)  did  not 
separate  or  identify  the  basic  presentation  statements,  the  feedback  (objectives 
3.1  and  3.2  only),  and  the  examples.  The  practice  using  was  separated  and 
identified  because  it  was  performed  separately  as  a  laboratory  exercise.  For 
example,  for  objective  3.1,  the  steps  of  the  Koga  procedure  were  given  along  with 
a  lot  of  nice-to-know  anecdotal  information.  The  steps  were  not  listed  separately 


on  the  blackboard  or  viewgraph  nor  were  they  identified  as  “step  1,  step  2,  etc.’’ 
The  practice  remembering  and  feedback  for  objective  3.3  were  separated  from  the 
other  instructional  components,  were  identified  and  clear,  but  the  practice 
remembering  for  objectives  3.1  and  3.2  was  not.  Despite  these  problems,  the 
statements,  examples,  and  practice  using  for  objectives  3.1  and  3.2  were  clearly 
presented  (column  3).  The  feedback  for  these  objectives,  as  noted  in  the 
consistency  evaluation  (Figure  4),  was  incomplete,  unclear,  and  not  separated  and 
identified.  Finally,  there  were  generally,  too  few  “helps”  but  the  instruction  was 
job  oriented  and  the  instructor  did  encourage  student  involvement.  In 
conclusion,  presentation  adequacy  in  this  course  could  be  significantly  improved. 
If  this  is  done,  it  would  enhance  student  learning  and  retention. 


Instructional  Effectiveness  Evaluation 

To  evaluate  the  instructional  process,  you  need  to  complete  the  Instructional 
Effectiveness  Checklist,  Appendix  E,  while  you  are  observing  the  classroom 
and/or  laboratory  instruction.  Figure  6  shows  the  completed  form  for  the 
classroom  observation  for  the  Master-at-Arms  course.  We  used  a  different  copy 
of  the  checklist  for  classroom  and  laboratory  instruction  because  the  types  of 
activities  were  different.  (Only  the  classroom  form  is  shown  in  Figure  6).  Be  sure 
to  label  the  laboratory  checklists  as  such  for  easy  identification. 

The  majority  of  checks  on  Figure  6  (Appendix  E)  are  favorable.  Some 
exceptions  are  that  the  instructor  did  not  urge  the  students  to  take  notes  nor  did 
he  give  the  students  time  to  take  notes.  Also  the  instructor  tried  to  teach  the 
class  without  any  overhead  transparencies  nor  other  aids  demonstrating  the 
proper  way  to  use  the  baton.  We  noted  that  this  aspect  of  the  instruction  could 
be  improved. 


Evaluation  Summary 

1.  Course  objectives  3.1  and  3.2  are  well  written  and  essential.  They  all  have 
appropriate  actions,  conditions  and  standards. 

2.  Objective  3.3  and  test  item  47  did  not  have  an  action  or  conditions  and 
standards  that  were  appropriate  for  the  job  requirements.  In  addition,  test  item 
46  did  not  match  the  objective.  The  performance  tests  for  objectives  3.1  and  3.2 
did  not  have  checklists/rating  scales  and  were  inadequate  because  there  was  too 
little  feedback  and  individual  supervision  during  the  the  laboratory  exercise. 

3.  Objectives  3.1,  3.2  and  3.3  were  adequately  introduced  to  the  students.  To 
little  time  seemed  to  be  devoted  to  practice  with  the  Use  objectives.  The 
instructor  provided  a  number  of  job  oriented  examples  for  all  objectives. 

4.  Presentation  Adequacy  evaluation  shows  that  instructional  components  were 
not  separated  or  identified.  It  also  shows  that  the  instructor  was  clear  and  job 
oriented  and  involved  the  students  in  the  presentation. 
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5.  The  Instructional  Effectiveness  Checklist  indicates  that  the  students  were  not 
encouraged,  nor  allowed  enough  time,  to  take  notes. 

6.  Given  the  number  of  students  in  the  laboratory  and  the  procedural  nature  of 
the  tasks,  using  a  baton,  films  or  videotapes  showing  the  correct  procedure  would 
have  been  appropriate.  No  visual  aids  were  used  by  the  instructor; 
demonstration  alone  was  used  to  show  the  correct  procedure. 


Course  Revisions 

1.  Test  item  46  should  be  eliminated  or  rewritten. 

2.  The  action  (identify),  standards  (70  percent)  and  conditions  (multiple  choice) 
for  objective  3.3  and  test  item  47  should  be  changed  so  that  the  item  is 
constructed  response  (e.g.,  listing)  with  100  percent  accuracy. 

3.  Checklists/rating  scales  should  be  developed  for  objective  3.1  and  3.2. 

4.  Additional  laboratory  time  should  be  allotted  to  provide  students  with 
increased  feedback  about  their  performance. 

5.  Audio-visual  aids  should  be  developed  to  illustrate  the  procedural  objectives. 

6.  Students  should  be  instructed  to  take  notes. 

7.  The  instructor  guide  should  be  revised  so  that  the  different  components  of  the 
instruction  will  be  separated  and  identified  by  the  instructor. 
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FIGURE  1 


LESSON  TOPIC  3.1  USE  AND  APPLICATION  OF  THE  POLICE  BATON 

Contact  Hours  Allotted  this  Lesson  Topic: 

Classroom  Laboratory 

0.5  Hours  2.5  Hours 

TERMINAL  OBJECTIVE: 

Supported  Entirely  by  this  Lesson  Topic:  NONE 

Supported  Partially  by  this  Lesson  Topic  and  Partially  by  Lesson  Topics  3.2  and  3.3: 
When  students  complete  this  course,  they  will  be  able  to: 

3.0  IDENTIFY  and  DEMONSTRATE  the  proper  use  of  physical  restraint,  self  defense, 
apprehension,  handcuffing  and  search  techniques  while  applying  the  concept  of 
Minimum  Force  in  accordance  with  applicable  publications. 

ENABLING  OBJECTIVES: 

Supported  Entirely  by  this  Lesson  Topic: 

When  students  complete  this  course,  they  will  be  able  to: 

3.1  Given  a  standard  police  baton  and  following  the  correct  steps  of  procedure, 
DEMONSTRATE  the  basic  movements  of  The  Koga  Method  of  Police  Baton 
Utilization  in  accordance  with  applicable  publications. 

3.2  DEMONSTRATE  the  correct  striking  areas  of  the  human  body  in  relationship 
to  the  use  of  the  standard  police  baton  and  the  levels  of  force  pertaining 

to  each  striking  area  in  accordance  with  applicable  publications. 

3.3  IDENTIFY  the  non  striking  areas  of  the  human  body  in  relationship  to  the 
use  of  the  standard  police  baton  as  specified  in  applicable  publications. 

A  score  of  70%  must  be  attained  on  a  final  written  comprehensive  test. 
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FIGURE  2 
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41.  HOW  IS  COCAINE  NORMALLY  ADM  IN  IS  Tl:  'ED? 

A.  SNIFFED 

B.  INJECTED 

C.  EATEN 

D.  BOTH  A  AND  C 

42.  MARIJUANA  DEPENDENCE,  IF  \NY,  IS  CLASSIFIED  AS? 

A.  REGULAR  USE  LEADS  TO  MENTAL  DEPENDENCE  ONLY 

B.  REGULAR  USE  LEADS  TO  PHYSICAL  DEPENDENCE  ONLY 

C.  REGARDLESS  OF  AMOUNT  OF  USE,  NO  DEPENDENCE  IS  DEVELOPED 

D.  CHRONIC  USERS  RISK  BODY  TOLERANCE  LEADING  TO  PHYSICAL  DEPEN¬ 
DENCE  IN  ADDITION  TO  MENTAL  DEPENDENCE. 

4?.  THE  GROUP  OF  DRUGS  OBTAINED  FROM  THE  POPPY  PLANT  ARE  CLASSIFIED 
AS? 

A.  HALLUCINOGENS 

B.  STIMULANTS 

C.  DEPREL'AN1 

d.  none  of  A?  ;  vl 

44.  WHAT  CO JRT-.M.  ART  -I.  CAN  AWARD  THE  GREATEST  PUNISHMENT? 

A.  CAPTAIN'S  MAST 

B.  SPECIAL  COURT-MARTIAL 

C.  SUMMARY  COURT-MARTIAL 

D.  GENERAL  COLE-  1 -MARTIAL 

45.  ONE  OF  THE  IMPORT aNt  PERSONAL  CHARACTER  TRAITS  OF  A  MASTER-AT- 
ARMS  IS  INTEGRITY;  INTEGRITY  IS  DEFINED  AS? 

A.  ENTERING  !;  .’TO  A  JOB  AS  A  PROFESSIONAL 

B.  A  KEEN  SEN:-:.  OF  WHAT  TO  DO  OR  SAY  TO  OTHERS 

C.  FIRM  ADHERENCE  TO  A  CODE  OF  ETHICS  (VALUES) 

D.  THE  QUALIFY  OF  BEING  JUST,  FAIR  AND  IMPARTIAL 

46.  WHEN  YOU  SEARCH  h  SUSPECT,  INCIDENT  TO  APPREHENSION,  YOU  MAY 
REMOVE  WHAT  FROM  THE  SUSPECT? 

A.  WEAPONS  ONLY 

B.  TOOLS  Of  THE  CRIME 

C.  FRUITS  Or  THE  CRIME 

D.  ANYTHING 

47.  WHAT  ARE  THE  THREE  AREAS  OF  THE  HUMAN  BODY  THE  MASTER-AT-ARMS 
WILL  NOT  STRIKE  WITH  THE  POLICE  BATON? 

A.  HEAD,  SPINE  AND  KNEE  CAPS 

B.  HEART,  SPINE  AND  HEAD 

C.  SPINE.  HEAD  AND  JOINTS 

D.  HEAD,  HEART  ND  SOLAR  PLEXUS 
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APPENDIX  b  objective  AW  TEST  REVIEW  ECRW 


FIGURE  3 
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FIGURE  4 
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PRESENTATION  CONSISTENCY  FORM 


Component 

Completeness 

Objective  # 

3.1 

3.2 

3.3 

1.  Statement 

C 

C 

c 

2.  Practice  Remembering 

c 

C 

c 

3.  Feedback 

s 

c. 

*1.  Examples 

c 

c 

y 

5.  Practice  Using 

c 

c 

6.  Feedback 

1 

J 

X 

C'  =  Componment  Complete 
I  =  Component  Incomplete 
X  =  Component  X'ot  Present 
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FIGURE  5 


Appendix  D 


PRESENTATION  ADEQUACY  FORMS 

I  Course  Title  /jf,  A  A  /J-f/o*  j_  3-sis c/o<l. _ 


Obj.#  3.  1 

(S3 

Iden. 

Clear 

Job 

Enough 

Errors 

E-H 

Invl 

1.  Statement 

A/ 

y 

y 

y 

y 

XXX 

XXX 

XXX 

Y 

2.  Practice  Remembering 

A/ 

V 

XXX 

y 

y 

'A 

XXX 
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PRESENTATION  ADEQUACY  FORMS 

Course  Title _  MA4  Af/.**  XjuJ*. 
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INSTRUCTIONAL  EFFECTIVENESS  CHECKLIST 

Instructions:  Answer  yes  or  no  to  relevant  questions;  do  not  answer  irrelevant 
questions  nor  force  inappropriate  yes  or  no  answers.  Use  the  back  of  the  form  to 
provide  additional  details  as  required.  Answer  the  following  questions  as  the 
lesson  progresses. 


LEARNING  ORIENTATION 

Yes 

No 

Are  students  given  objectives? 

Were  the  students  motivated  in  terms  of  “why”  the  content 
should  be  learned? 

1/ 

Were  the  students  motivated  re  “how”  the  content  should 
be  used? 

1/ 

Are  students  told  how  they  will  be  tested? 

Are  students  told  what  they  will  be  tested  on? 

Did  instructor  frequently  orient  students  to 
job  of  learning?  Was  information 
about  the  instructional  process  and  what  the 
students  were  supposed  to  be  doing/learning  given? 
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FIGURE  6  (continued) 

Instructional  Effectiveness  Checklist  (Continued) 


INSTRUCTOR  BEHAMOR 


Was  peer  instruction  used 


Were  external  rewards  given 


Did  instructor  establish  relationship  with  students  by  introducing 
her/himself? 

by  displaying  course/unit/module  name? 
by  creating  interest  in  subject? 
by  displaying  enthusiasm? 

by  soliciting  class  cooperation  and  involvement? 


Is  the  instructor's  voice  level  and  enunciation  adequate' 


Was  the  instructor  free  from  any  distracting  mannerisms 
(e.g. ticks,  twitches)’ 


Did  it  appear  that  the  instructor  adequately  prepared  for 
the  lesson? 


Did  the  instructor  urge  students  to  take  notes? 


At  appropriate  points  in  lecture,  did  the  instructor 
pause  and  indicate  to  students  thev  should  take  notes? 


Did  Instructor  allow  students  to  learn  by 
doing  when  possible? 


Did  instructor  ensure  that  students  were  activelv  involved 


Did  instructor  monitor  student  progress? 


Did  instructor  provide  assistance  when  necessary? 
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FIGURE  6  (continued) 

Instructional  Effectiveness  Checklist  (Continued) 


INSTRUCTOR  BEHAVIOR  (CON’T) 

Yes 

No 

Was  IG  used  properly? 

Was  Curriculum  Outline  adhered  to? 

l/ 

Was  student  comprehension  checked? 

S 

Were  proper  questioning  techniques  used? 

s 

Was  class  control  maintained? 

s 

Was  eve  contact  maintained? 

s 

Was  summary /critique  appropriate? 

i 
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FIGURE  6  (continued) 

Instructional  Effectiveness  Checklist  (Continued) 


MEDIA  /  MATERIALS 

Yes 

No 

What  tvpes  of  media  were  used?  C'/)4f  K 

X 

X 

Were  they  appropriate? 

V 

Were  the  transparencies, etc.,  technically  correct 
(easv  to  read,  layout  attractive,  etc.) 

1/ 

Were  there  sufficient  number  of  instructional  aids? 

(example,  could  transparencies  be  used  instead 
of  blackboard?) 

V 

Were  they  effectively  used  in  separating  the  various 
instructional  activi ties /phases /summary? 

✓ 

ENVIRONMENT  /  SAFETY 

Describe  significant  environmental  factors  involving 
light,  background  noise,  temperature,  general 
attractiveness  of  classroom,  etc. 

X 

X 

Was  the  equipment  operational? 

V4 

Were  necessary  tools  and  test  equipment  available? 

Was  lab  clean  and  free  of  safety  hazards? 

S'* 

Was  two-man  rule  observed? 

S'* 

Were  safety  precautions  announced? 

P4 

Were  students  aware  of  emergency  procedures? 

V* 

Was  emergency  first  aid  procedure  posted 
and  visible? 

H 

Were  high  voltage  areas  clearly  marked? 

S'* 
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FIGURE  6  (continued) 

Instructional  Effectiveness  Checklist  (Continued) 


STUDENT  BEHAMOR 

Yes 

No 

Did  it  appear  the  students  achieved  the  objectives? 

Did  it  appear  the  students  took  adequate  amount  of 
notes? 

V 

How  involved  were  the  students  in  the  learning 
process0  Circle  the  most  appropriate  category 
listed  below  that  describes  the  majority  of 
the  students. 

ACTPE  Students  ask  questions,  answer 

INVOLVEMENT  questions,  volunteer  extra  information 
generate  information.  May 
go  beyond  information  immediateK  given 
and  relate  to  other  known  information. 

/^ATTENTTVE^V Students  clearly  understand  and  are  tracking 

V  INVOLVEMENT^  what  teacher  is  saying.  Seem  interested. 
Indicated  by  alert  faces,  heads  nodding 
positively,  alert  posture,  smiling,  talking 
to  selves. 

✓ 

PASSIVE  Students  seem  bored  and  are  only  marginally 

INVOLVEMENT  paying  attention  to  what  is  going  on  in  the 
class.  Are  able  to  answer  straighforward 
questions,  but  do  not  exhibit  a  great 
deal  of  interest  in  what  is  being  taught. 

✓ 

NO  Students’  eyes  may  be  glazed  over  and  may 

INVOLVEMENT  have  difficulty  staying  awake.  Can’t  answer 
question  if  called  upon.  May  not 
even  seem  to  know  what  lesson  instructor 
is  addressing. 

A- 18 


APPENDIX  B  OBJECTIVE  AND  TEST  REVIEW  FOR* 


(HJdSNVdl)  H9HON3 
3i\/n03av  1S31 
•dOdddV  1S31 

HOiVW  NO I 13V 
HD1VW  OMNVIS 

HOIVW  NOIliaNOO 

n  IJ31 1  1S31 

no i 10 v  rao 
aavaNVis  rao 
NOiiiaNOO  rao 

13A33  ONINIV'dl 
AM\/SS303NNn 
MON>I-Oi-30IN 
1VUN3SS3 
rao  ddns 

13A31  MSVl 

#  rao 


CO 

ex 

=5 

o 


«x 

o 

o 

u? 


<c 

ex 


B-l 


1 


Appendix  D 


PRESENTATION  ADEQUACY  FORMS 

Course  Title _ 


Obj.# 
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INSTRUCTIONAL  EFFECTIVENESS  CHECKLIST 

Instructions:  Answer  yes  or  no  to  relevant  questions;  do  not  answer  irrelevant 
questions  nor  force  inappropriate  yes  or  no  answers.  Use  the  back  of  the  form  to 
provide  additional  details  as  required.  Answer  the  following  questions  as  the 
lesson  progresses. 


LEARNING  ORIENTATION 

Yes 

No 

Are  students  given  objectives? 

Clarified/amplified  (if  necessary)? 

Were  the  students  motivated  in  terms  of  “why’-  the  content 
should  be  learned0 

Were  the  students  motivated  re  “how”  the  content  should 
be  used? 

Are  students  told  how  they  will  be  tested0 

Are  students  told  what  they  will  be  tested  on? 

Did  instructor  frequently  orient  students  to 
job  of  learning?  Was  information 
about  the  instructional  process  and  what  the 
students  were  supposed  to  be  doing/learning  given? 

E-l 


Instructional  Effectiveness  Checklist  (Continued) 

INSTRUCTOR  BEHAVIOR 


Yes 


No 


Did  instructor  establish  relationship  with  students  by  introducing 
her/himself’’ 

by  displaying  course/unit/module  name? 
by  creating  interest  in  subject? 
by  displaying  enthusiasm? 

by  soliciting  class  cooperation  and  involvement? _ 

Is  the  instructor's  voice  level  and  enunciation  adequate? _ 

Was  the  instructor  free  from  any  distracting  mannerisms 
(e.g. ticks,  twitches)? _ 

Did  it  appear  that  the  instructor  adequately  prepared  for 
the  lesson? 


Did  the  instructor  urge  students  to  take  notes? _ 

At  appropriate  points  in  lecture,  did  the  instructor 
pause  and  indicate  to  students  they  should  take  notes? 

Did  Instructor  allow  students  to  learn  by 
doing  when  possible? _ 

Did  instructor  ensure  that  students  were  actively  involved? 

Did  instructor  monitor  student  progress? _ 

Did  instructor  provide  assistance  when  necessary? _ 
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Instructional  Effectiveness  Checklist  (Continued) 


INSTRUCTOR  BEHAVIOR  (CQN'T) 
Was  IG  used  properly? _ 

Was  Curriculum  Outline  adhered  to? 


Was  student  comprehension  checked?  _ 

Were  proper  questioning  techniques  used? 
Was  class  control  maintained? 


Was  eye  contact  maintained? 


Was  summary/critique  appropriate? 


Instructional  Effectiveness  Checklist  (Continued) 


MEDIA  /  MATERIALS 


What  tvpes  of  media  were  used 


Were  the  transparencies, etc.,  technically  correct 
(easy  to  read,  lavout  attractive,  etc. 


Were  there  sufficient  number  of  instructional  aids? 
(example,  could  transparencies  be  used  instead 
of  blackboard?) 


Were  they  effectively  used  in  separating  the  various 
instructional  activities/phases/summary? 


ENVIRONMENT  /  SAFETY 


Describe  significant  environmental  factors  involving 
light,  background  noise,  temperature,  general 
attractiveness  of  classroom,  etc. 


Was  the  equipment  operational? 


Were  necessary  tools  and  test  equipment  available? 


Was  lab  clean  and  free  of  safety  hazards?  _ 


Was  two-man  rule  observed 


Were  safety  precautions  announced? 


Were  students  aware  of  emergency  procedures? 


Was  emergency  first  aid  procedure  posted 
and  visible? 


Were  high  voltage  areas  clearlv  marked? 


Yes 

No 

X 

X 

E-4 


Instructional  Effectiveness  Checklist  (Continued) 
STUDENT  BEHAVIOR 


Did  it  appear  the  students  achieved  the  objectives? 

Did  it  appear  the  students  took  adequate  amount  of 
notes? _ 

How  involved  were  the  students  in  the  learning 
process?  Circle  the  most  appropriate  category 
listed  below  that  describes  the  majority  of 
the  students. 


ACTIVE  Students  ask  questions,  answer 
INVOLVEMENT  questions,  volunteer  extra  information 
generate  information.  May 
go  beyond  information  immediately  given 
and  relate  to  other  known  information. 


ATTENTIVE  Students  clearly  understand  and  are  tracking 
INVOLVEMENT  what  teacher  is  saying.  Seem  interested. 
Indicated  by  alert  faces,  heads  nodding 
positively,  alert  posture,  smiling,  talking 
to  selves. 


PASSIVE  Students  seem  bored  and  are  only  marginally 
INVOLVEMENT  paying  attention  to  what  is  going  on  in  the 
class.  Are  able  to  answer  straighforward 
questions,  but  do  not  exhibit  a  great 
deal  of  interest  in  what  is  being  taught. 


NO  Students’  eyes  may  be  glazed  over  and  may 
INVOLVEMENT  have  difficulty  staying  awake.  Can't  answer 
question  if  called  upon.  May  not 
even  seem  to  know  what  lesson  instructor 
is  addressing. 
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